Executive Summary 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 



SCALING 
UPTHE 
SUCCESS 
FOR ALL 
MODEL OF 
SCHOOL 
REFORM 

Final Report from the 
Investing in Innovation 
(i3) Evaluation 


Janet Quint 
Pei Zhu 
Rekha Balu 
Shelley Rappaport 
Micah DeLaurentis 

September 2015 





Executive Summary 


Scaling Up the Success for All Model 
of School Reform 

Final Report from the 
Investing in Innovation (i3) Evaluation 


Janet Quint 
Pei Zhu 
Rekha Balu 
Shelley Rappaport 
Micah DeLaurentis 

with 

Emma Alterman 
Colin Bottles 
Emily Pramik 


September 2015 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 



Funding for this report came from the U.S. Department of Education under its Investing in 
Innovation (i3) initiative. The i3 grant called for the Success for All Foundation to scale up its 
elementary school whole-school reform initiative and for MDRC to conduct an independent 
evaluation of the implementation and impacts of that expansion. 

Dissemination of MDRC publications is supported by the following funders that help finance 
MDRC’s public policy outreach and expanding efforts to communicate the results and implica¬ 
tions of our work to policymakers, practitioners, and others: The Annie E. Casey Foundation, 
Charles and Lynn Schusterman Family Foundation, The Edna McConnell Clark Foundation, Ford 
Foundation, The George Gund Foundation, Daniel and Corinne Goldman, The Harry and Jeanette 
Weinberg Foundation, Inc., The JBP Foundation, The Joyce Foundation, The Kresge Foundation, 
Laura and John Arnold Foundation, Sandler Foundation, and The Starr Foundation. 

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. Con¬ 
tributors to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell Foundation, 
An heuser-Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart Mott Founda¬ 
tion, Ford Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth and 
Frank Newman Charitable Foundation, The New York Times Company Foundation, Jan Nichol¬ 
son, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler Foundation, and The Stupski 
Family Fund, as well as other individual contributors. 


The findings and conclusions in this report do not necessarily represent the official positions or 
policies of the funders. 


For information about MDRC and copies of our publications, see our website: www.mdrc.org. 
Copyright © 2015 by MDRC®. All rights reserved. 



Overview 


Success for All (SFA) is one of the best-known school refonn initiatives. Combining a chal¬ 
lenging reading program, whole-school refonn elements, and an emphasis on continuous im¬ 
provement, it seeks to ensure that every child leams to read well in the elementary grades. In 
2010, the Success for All Foundation (SFAF) received a scale-up grant under the U.S. Depart¬ 
ment of Education’s Investing in Innovation (i3) program. This third and final report from the 
independent evaluation of the i3 scale-up examines the program’s implementation and impacts 
over three years, its incremental cost, and the scale-up process itself. Thirty-seven evaluation 
schools in five school districts were randomly assigned either to a program group of 19 schools 
that received SFA or to a control group of 18 schools that used alternative reading programs. 
This design supports causal impact findings for the average school assigned to SFA. Overall, 
the evaluation led to several key findings: 

• Although SFA was implemented with adequate fidelity at the great majority of schools that 
adopted it, resource constraints prevented some schools from putting in place some of its 
key features, including a full-time facilitator and SFA’s computerized tutoring program. 

• Program group and control group schools were different in some respects (for example, 
SFA schools were unique in placing students in cross-grade ability groups for reading, and 
SFA teachers made greater use of cooperative learning) but similar in others. 

• SFA is an effective vehicle for teaching phonics. In the average SFA school, the program 
registered a notable, statistically significant impact on a measure of phonics skills for sec¬ 
ond-graders who had been in SFA for all three years, compared with their control group 
counterparts. Students in the average SFA school did not outperform their counterparts in 
the average control group school on tests of reading fluency or comprehension. 

• For a subgroup of special concern to policymakers and practitioners — students entering 
school with low preliteracy skills — SFA appears to be especially effective. Second-graders 
in the average SFA school who had started kindergarten in the bottom half of the sample in 
terms of their knowledge of the alphabet and their ability to sound out words registered sig¬ 
nificantly higher scores on measures of phonics skills, word recognition, and reading fluen¬ 
cy than similar students in control group schools. The impact on comprehension for this 
group was also positive but not statistically significant. The program did not significantly 
affect outcomes for the subgroup of students who started kindergarten in the top half of the 
sample in terms of phonetic skills. 

• In a case study district, the direct expenditures for additional reading facilitator time, after¬ 
school tutoring, materials, and professional development were estimated to cost $119 more 
per student per year in SFA schools than in control group schools. Including the extra time 
that SFA principals devoted to the program and that coaches and teachers spent in training, 
the extra cost of space for storing SFA materials, and other factors, program group schools 
spent about $227 worth of resources per student per year more than control group schools. 

• Through the fourth year of the i3 grant, SFA was put in place in 447 new schools and 
reached an estimated 276,000 students. These numbers fell below SFAF’s ambitious goals 
but represent a notable achievement in a period of staff layoffs and other cutbacks in many 
schools and districts. 
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Preface 


Low reading skills remain a pressing problem in the United States. With the emergence of more 
demanding academic expectations — as represented by the Common Core State Standards — 
students will need stronger reading skills to master more complex content. Despite more than a 
decade of concentrated effort to improve reading instruction, elementary student reading 
achievement scores on the National Assessment of Educational Progress (“The Nation’s Report 
Card”) have increased only modestly, signaling the need for continued improvement. The Suc¬ 
cess for All Foundation (SFAF) has been active in reading instruction for some three decades, 
building evidence of the effectiveness of its approach in improving the reading skills of elemen¬ 
tary school students. Thus, it was not a surprise that the Success for All reading program was 
one of the initial recipients of a federal Investing in Innovation (i3) scale-up grant designed to 
provide support for the expansion of evidence-based programs. This report, the last in a series of 
three, completes MDRC’s evaluation of the SFA elementary reading program under i3. 

SFAF works with schools to put in place a comprehensive approach to reading instruc¬ 
tion. It involves a highly structured curriculum that covers the five key elements identified by 
the National Reading Panel (phonemic awareness, phonics, fluency, vocabulary, and compre¬ 
hension); a strong emphasis on cooperative learning; cross-grade grouping of students by read¬ 
ing skills; extensive use of assessment data to adjust groupings and instruction; and tutoring for 
students in need of additional help. SFA also includes schoolwide structures that address attend¬ 
ance, behavior, and parental involvement to support student learning. Full implementation of 
this comprehensive approach has required substantial effort on the part of school staff already 
facing the challenges of funding cuts resulting from the recession as well as the new demands of 
the Common Core standards. Thus, the scale-up of SFA under i3 was especially ambitious, but 
also timely in light of current efforts to encourage evidence-based funding decisions and the 
expansion of proven programs. 

The report provides findings on all aspects of the SFA reading scale-up initiative: Was 
the program implemented with fidelity as it was scaled up? Did SFA produce a difference in 
reading instruction compared with instruction in business-as-usual schools in the study? Did 
schools implementing SFA have better student achievement than alternative reading programs, 
on average and for key subgroups of students? What were the extra costs of implementing 
SFA? And how did SFA fare in its efforts to increase the number of schools implementing the 
program in a time of severe economic constraints? The answers to these questions are highly 
relevant as policymakers strive to increase the use of evidence in decision-making. 

Gordon L. Berlin 

President, MDRC 


v 




Acknowledgments 


This third and final report of the Investing in Innovation (i3) scale-up evaluation of Success for 
All reflects the efforts of a great many people. As with the two previous reports, our first debt of 
gratitude is to the principals, SFA coordinators, research liaisons, and teachers at the schools 
that participated in this study; to staff members in the central offices of the five school districts 
who provided us with critical student records data as well as fiscal and staffing information; and 
to district coaches who took part in interviews. The assistance and cooperation of these individ¬ 
uals were vital for enabling the study to go forward and for providing the rich and detailed in¬ 
formation on which this report is based. 

At the Success for All Foundation, Nancy Madden, Sharon Fox, and Jill Hanson re¬ 
sponded promptly and patiently to our many information requests. Nancy Madden and Robert 
Slavin provided useful critiques at each stage of the report drafting process. 

Pamela Wells and her capable team at Decision Information Resources, Inc., produced 
the student assessment data that are at the heart of the impact study. MDRC staff members 
Jo Anna Hunter and Nicole Morris worked with DIR to ensure that these data were collected on 
schedule and with minimal disruption to school operations. Thomas J. Smith played an active 
role in field research, data collection, and coding. 

Within MDRC, Seth Muzzy helped prepare the principal and teacher surveys and the 
instructional logs that constitute important data sources for the report. Shirley James, Zuleka 
Abakoyas, Donna George, and Carmen Troche keyed surveys and logs. Emma Alterman, Colin 
Bottles, Cammie Brown, Zachary Pinto, Emily Pramik, and Gary Reynolds played important 
roles in data collection, analysis, and the preparation of this report. 

Fred Doolittle ensured that the team received both material and moral support at every 
juncture. He, along with Gordon Berlin, Howard Bloom, Charles Michalopoulos, Jean Gross- 
man, Robert Ivry, and Leigh Parise, carefully reviewed earlier drafts of the report and made 
comments that improved the final product. Kelly Granito kept the project on task and on budget, 
and Gina Price provided general fiscal oversight. Mario Flecha was his usual unflappable self in 
handling a variety of administrative and other tasks. Christopher Boland provided advice on 
preparing exhibits, and Stephanie Cowell prepared the manuscript for publication. 

Jennie Kaufman edited the report with great sensitivity, and John Hutchins proffered 
welcome assistance as report production drew to a close. 


The Authors 




Executive Summary 


In 2013, almost one-third of fourth-grade students in the United States scored below the “basic” 
level in reading, according to the National Assessment of Educational Progress (NAEP), also 
known as “The Nation’s Report Card.” 1 They could not locate relevant infomiation in a text, 
make simple inferences based on what they read, or identify details to support an interpretation 
or conclusion. Students perfonning at this level lack the skills needed to demonstrate solid aca¬ 
demic performance or to master challenging subject matter. 

Success for All (SFA), one of the best-known school reform models, aims to improve 
the reading skills of all children but is especially directed at schools that serve large numbers of 
students from low-income families. First implemented in 1987, SFA combines a challenging 
reading program, whole-school reform elements, and an emphasis on continuous improvement, 
with the goal of ensuring that every child leams to read well in the elementary grades. SFA in¬ 
cludes several specific features: 

• A kindergarten through grade 6 reading program that emphasizes phonics for 
beginning readers and comprehension for all students 

• Instruction that is characterized by “scripted,” briskly paced lesson plans that 
make extensive use of cooperative learning in pairs and small groups 

• Cross-grade ability grouping for reading, with many students leaving their 
homeroom to receive reading instruction from another teacher, and quarterly 
regrouping 

• Frequent assessments of student learning 

• Computerized small-group tutoring and individual tutoring for students who 
need additional assistance 

• Staff committees (“Solutions Teams”) that address academic, behavior, and 
attendance issues and that promote parent and community involvement 

• Schoolwide and classroom programs to develop social and conflict resolution 
skills 


'See National Assessment of Educational Progress, “Nation’s Report Card: 2013 Mathematics and Read¬ 
ing’’ (2013), http://nationsreportcard.gov/reading_math_2013. The percentage of students performing below 
the basic level was 32 percent. NAEP is a congressionally authorized project of the National Center for Educa¬ 
tion Statistics within the Institute of Education Sciences in the U.S. Department of Education. NAEP tests have 
been conducted periodically in a number of subject areas since 1969. 
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Initial and ongoing professional development for teachers and use of data to 
monitor progress and set goals 


This is the third and final report from an independent evaluation of the scale-up demonstration 
of the SFA elementary school reading program. Both the demonstration and the evaluation have 
been funded under the U.S. Department of Education’s Investing in Innovation (i3) competi¬ 
tion. Conducted by MDRC — a nonprofit, nonpartisan education and social policy research 
organization — the evaluation examines SFA’s implementation and impacts in five school dis¬ 
tricts over a three-year period (the 2011-2012 school year through the 2013-2014 school year). 
It also includes an analysis of program costs. Finally, it considers the scale-up process itself— 
the methods employed and the extent to which the Success for All Foundation (SFAF), the or¬ 
ganization that developed and provides technical assistance to schools operating the program, 
achieved its scale-up goals. 

Previous evaluations, both experimental and quasi-experimental, showed that students 
in SFA schools performed better on standardized tests than students receiving other reading 
programs. The most salient of these evaluations was a three-year randomized experiment in¬ 
volving 35 schools serving low-income families. In that study, schools were randomly assigned 
to use Success for All either in kindergarten through grade 2 (K-2) or grades 3 through 5, with 
the schools that received the program in the later grades serving as a control group for the K-2 
schools. 2 Children in the K-2 schools scored significantly higher than their counterparts in the 
3-5 schools on the main outcomes measured — three tests that assessed children’s phonetic 
skills and comprehension. In other large-scale studies, results for students in SFA schools out¬ 
stripped those for students in matched comparison schools. 3 The strength of this evidence was 
critical to the selection of the Success for All Foundation as one of only four recipients of five- 
year scale-up grants awarded in 2010 in the initial i3 funding competition. 


The Evaluation Design 

The i3 evaluation of SFA employs an experimental design, in which 37 schools in five school 
districts that participated in the scale-up effort were assigned at random to a program group or to 
a control group. The 19 program group schools received SFA in all grades. The 18 control 
group schools did not get the intervention and, instead, either continued with the same reading 


2 See Geoffrey D. Borman, Robert E. Slavin, Alan C. K. Cheung, Anne M. Chamberlain, Nancy A. Mad¬ 
den, and Bette Chambers, “Final Reading Outcomes of the National Randomized Field Trial of Success for 
All,” American Educational Research Journal 44, no. 3 (2007): 701-731. 

3 See, for example, Brian Rowan, Richard Correnti, Robert J. Miller, and Eric M. Cambum, School Im¬ 
provement by Design: Lessons from a Study of Comprehensive School Reform Programs (Philadelphia: Con¬ 
sortium for Policy Research in Education, 2009). 
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program that they had used previously or, in the case of some schools, adopted a new one. This 
design supports causal impact findings for the average school assigned to SFA. 


Context for the Evaluation 

It is useful to consider the economic and instructional contexts in which the SFA scale-up 
demonstration has unfolded. These contexts provide a framework through which to view the 
participating schools’ ability to implement the full program model and SFAF’s ability to meet 
its ambitious expansion goals. They also help to define the “counterfactual” — what happens in 
the absence of the program. Only to the extent that SFA differs from the counterfactual is the 
program likely to produce impacts. Two trends are worth noting: 

• The effects of the Great Recession and its aftermath. At the point that 
SFAF was recruiting schools for the i3 scale-up, many schools and districts 
were trying to restore positions and services that had been cut as a result of 
the recession. Furthermore, principals felt that they had less discretion in 
spending their schools’ allocations than had been the case in the past. These 
circumstances added a new dimension to the challenges already associated 
with selecting and implementing a new and demanding reading program in 
high-poverty schools. 

• Heightened focus on reading instruction. Over the period since SFA was 
first developed in 1987, reading instruction in the United States has changed 
markedly. For example, the influence of the National Reading Panel report of 
2000, the passage of No Child Left Behind in 2001 (and, as a result, the crea¬ 
tion of the Reading First program and the advent of high-stakes testing for 
grades 3 through 8), the rise of Response to Intervention reading support 
strategies, and the introduction of the Common Core standards have all con¬ 
tributed to an increased emphasis on phonics and additional interventions for 
struggling readers. These developments have had the effect of narrowing the 
differences between schools adopting SFA and schools using other reading 
programs and have made it harder than it used to be for SFA to “beat the 
competition.” 


Findings 

Implementation 

While Success for All was implemented with adequate fidelity at the large majority of 
schools that adopted it, resource constraints prevented some schools from putting in place some 
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of the program’s key features, including a full-time program facilitator and SFA’s own comput¬ 
erized tutoring for students in need of instruction beyond the classroom. Reading instruction in 
program group schools (or “SFA schools”) and control group schools was markedly different in 
some respects: Placing students by ability level in reading groups that crossed grade levels was 
unique to the program group schools, and program group teachers used cooperative learning as 
an instructional method more frequently than their counterparts in the control group schools. In 
other respects, the two groups of schools did not differ greatly. Despite some implementation 
challenges, 93 percent of the principals and 70 percent of the teachers in SFA schools who re¬ 
sponded to surveys agreed that the SFA program benefited their schools. 

Effects on Phonics 

The evidence indicates that SFA is an effective vehicle for teaching phonics. In the av¬ 
erage SFA school, the program registered a positive and statistically significant third-year im¬ 
pact on a strong measure of phonetic abilities (one that asks students to read phonetically regu¬ 
lar nonsense words) for second-graders who had been in SFA classrooms for all three years, 
compared with their counterparts in control group schools — the groups that make up the “con- 
finnatory sample” for the i3 study. This effect was also found in the previous two years of the 
demonstration, when these students were kindergartners and first-graders. In the second year, 
SFA also produced a positive and statistically significant effect on another measure of phonetic 
skills; in Year 3, this effect remained positive but was no longer statistically significant. 

Effects on Comprehension 

In Year 3 as in previous years, in the average SFA school, students did not outperform 
their counterparts in the average control group school on measures of reading fluency or com¬ 
prehension. The comprehension finding is true for second-grade students in the confirmatory 
sample as well as for students in the upper elementary grades, for whom the analysis is consid¬ 
ered exploratory. (The comprehension finding contrasts with that of the previously cited exper¬ 
imental study of SFA, which found a positive and statistically significant effect on comprehen¬ 
sion for second-graders, as well as with several well-regarded quasi-experimental studies that 
found positive although not statistically significant effects.) 

Effects on Students with Low Preliteracy Skills 

Students who start school with low preliteracy skills are of special concern to policy¬ 
makers and practitioners. An exploratory analysis indicates that the program had notable third- 
year impacts on a subgroup of second-graders who, at the start of kindergarten, scored in the 
bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound 
out words. In the average SFA school, the program produced positive and statistically signifi- 
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cant impacts on measures of phonics skills, word recognition, and reading fluency for these stu¬ 
dents. The impact on comprehension was also positive, although it fell shy of meeting conven¬ 
tional standards of statistical significance. The program did not significantly affect these out¬ 
comes for the subgroup of students who started kindergarten in the top half of the sample in 
terms of phonetic skills. 

Effects on Special Education and Grade Retention Rates 

The program did not affect the rates at which students were held back to repeat a grade 
or at which they were identified for or declassified from special education. 

Cost Analysis 

The cost analysis makes use of the random assignment design within one case study 
district to assess the extent to which the district’s SFA schools required additional resources to 
implement the program, relative to those used for alternative reading programs in the control 
group schools. In the study district, the direct expenditures for school-based reading facilitator 
time, after-school tutoring time, materials, and professional development were estimated to cost 
$119 more per student per year in SFA schools than in control group schools. Adding to this the 
additional time that SFA principals devoted to the program, the additional time that coaches and 
teachers spent in training, the extra cost of devoting space to storing SFA curriculum materials, 
and other factors, program group schools spent about $227 worth of resources per student per 
year more than control group schools to implement their respective reading programs. 

Scale-Up 

During the first four years of scale-up, SFA was put in place in 447 new schools with a 
total enrollment of some 218,000 students and, taking into account student turnover, is estimat¬ 
ed to have reached some 276,000 students. While these numbers fell below the Success for All 
Foundation’s ambitious initial goal of recruiting 760 schools within the first four years, they 
represent a notable achievement, especially in a period when many schools and districts were 
laying off staff and cutting back on programs. 


Conclusion 

The i3 scale-up has heightened the prominence of Success for All on the educational landscape. 
In an economic climate characterized by budgetary cutbacks that forced many school districts to 
cut staff and restrict program offerings, the Success for All Foundation projects that it will have 
reached almost 400,000 students in 540 schools by the end of the i3 grant. The scale-up findings 
show that, for a modest investment, SFA reliably improves the decoding skills of students in 
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kindergarten through second grade, and that it is especially beneficial for students who begin in 
the lower half in these skills. 

Continuous improvement is a key element of the Success for All program. With a 
greater focus on improving comprehension and broader implementation of its tutoring compo¬ 
nent, Success for All might make an even bigger difference, and for more students, than it al¬ 
ready does. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of 
social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) 
and evaluations of ongoing government and community initiatives. MDRC’s staff bring an 
unusual combination of research and organizational experience to their work, providing 
expertise on the latest in qualitative and quantitative methods and on program design, 
development, implementation, and management. MDRC seeks to learn not just whether a 
program is effective but also how and why the program’s effects occur. In addition, it tries to 
place each project’s findings in the broader context of related research — in order to build 
knowledge about what works across the social and education policy fields. MDRC’s findings, 
lessons, and best practices are proactively shared with a broad audience in the policy and 
practitioner community as well as with the general public and the media. 

Overthe years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local 
governments, public school systems, community organizations, and numerous private 
philanthropies. 


